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^ PREFACE ' ' . ' . ..^ 

'; ' ; ■ ' 

The Research on Evaluation Program is a Nort-hwest: Regional Educational 
'"Laboratory project of research, development, * testing, and training 
'designed to create- n^w evaluation methodologies for use in education. 
, Thi3 .dpcumant is one: ojc' a series of papers *and rep*^ort3> produced; by 

program staff,, visiting scholars, adjunct , scholars, and' project 

collaborators— all members of a cooperative network of colleagues 

workifig on the development of new metho.dologies^ 

There are 21v675 docments tn ERIC descriptored "Evaluation". Can 
ERIC system features be used to pre-sort and categorize this litera- 
ture' by computer? How well does ERIC, cover the evaluation literature.? 
This report describes ERIC limitations whi'ch prevent computerized ; 
pre-9creening of literature,' and recounts the. failure of ERIC to 
reQoua^SUCt an existing evaluation bibliography (only L. 2 percent of 
the ite.ms w.ere identified by ERIC), casting serious doubt on tlie 
utility of ERIC searches 'bf the evaluation literature. 

■ ' • ' ' ^ " ... " -Nick Lo Smith, Editor 

Paper an'd Report Series _ 




•THE LIMITATIONS OF ERJC IN REVIEWING 
' -'^ EVALUATION LITERATURE^ 

It has become a counnon observatioji that educat;'ion£ll 'researchets ' 
now produce studies and reports at a" much faatV/rate ' than- can be read 
and understood, even by colleagues. The engulf ing.> literature long ago 
^surpassed the individual researcherjs- capacity to absorb nonintegrated 

•5 • .. • • • • . . • 

£in(tings and as a pfbfSSSion '\_we find ourselves in the mildly embarraU . 

; ■ . ■» 

sing position of knowing less ^than we have ^p^royen. (Glass,: 1976, p: ^8) 

' ■ s ■ ' \ ■ ' " ■ ' ' ■. * , 

/We recognize with Glass that ".•'.the armchair literature review iri which 

, .J , • ■ f 

i.one cites a couple dozen studies from the obvious*' journals can' t do 

justice to the voluminous literature of educational research that we now 

confront." ,(1976, p. 4) He and others (e.g., Light and Wraith, . 1971) have 

therefore stressed ^the'^need for integrative,^ scholarly revi^9 and have 

suggested procedures for cbmbining and interpreting the findings' frt>m mxiltiple 
- . . / . ,' ' ' ■ * ■ , ' ^ '1 

jstudies, ^ile these authors have dealt with the complex problems of inte- - 

^ r . ' ^ ' 

grating convergenii'data, I am cfoncerned here with the more mundane problem 

of how. we initiaJLly Identify which studies to review. The old method of" 

'screening several hundred studies by hand is no longer feasible when the 

potentially relevant literature contains thousands, not hundreds, of 

y 

citations, For^ example, we recently i>egan^ review of the literature on 
the raetJi^^dology of program evaluation and found- that in the ERlf (Educational* 
Rejsources «ftiformation. Center) syste'm: ' ' ' 

21 ,675 documents were descriptdred ''I^^aluation, " 
.13,433 documents were descripto.red "Program Evaluation," 
and 6,689 (jo.cuments were descriptored 'Evaluation Methods." 



. The Limitations 

•Since there were/ obviously too matiy documents to review by hand, we tried 
to use. certain features of the .ERIC system to reduce the collection. 

Using the £RIC interactive computer search capability, we ' 
attempted to identify and characterize • the literature of interest in 
our search. Our purpose was to isolate the "core" literature 
a'nd to reduce the number qf citations to a manageable size of 
about 2,000'. At that point we could print and screen 2,000 
abstracts by .hand, selecting a lesser number of . documents for Wtual 
study. As a result ^of dur efforts, we* 'were able to make some general 
assessments of the limitations of the ERIC System in conducting such • 
\ searches. t ' \^ ' , 

( • ^ , , . ■ 

f Chara'ctertzing Literature. Using ERIC ^ 

' ft. 

One of »the biggest prdblems in the early stages of an integrative 
literature review is to characterize the general 'nature of the relevant 
literature. This is a^ crucial step/ since subsequent screening and * 
review decisions are based', in part, on assumptions about the nature of 
the documents one is dealing with* ' ^ 

After ^isolating, what appeared to be' the relevant docunjents, we 
attempted ^to^ use ERIC to characterize their napure. Each document in- ^ 
ERIC ^ is labeled' by otultiple descriptor words which can be used to bi^ild 
document collections. Each documefit is given a mjLximum of five major 
descriptors and as many minor descriptors as necessary (up through 1972, 
, documents were given an average of 11 descriptors each (ERIC, 197A, p. 187).) 



■J^' ^ • The Limitations' 




3.S 



... . it'-^ 

To build an Initial document -collection we first combined all> 

those document^- r.elated to' program evaluation, broadly const^'ued , ' by 

using the. following descrip^tors : ' ^ ; 

Program Evaluation * (13,433 documents) ' / 

. - . ' - . - . . r - ^ / 

Curriculum Evaluation' - (2,139) 



\ " Formative EValu^tioti- (680) 



^ ' , Summative Evaluation ^ ' * (595) 

Medical E^/aluation ' (398) \ " ' 

^Medical Cara Svaluation' (34) 

^ . ■ . ^ - - • ^ ^ . \ 

, r Qental 'Evaluation . (27) t " ~ > 

This resulted in a ,tatal of 16,206 separate documents. In order ^o 

idfentify only the "core" or most significant, program- Evaluation c^ocyjments,* 

this" collection- was restricted to only tho'se using the above terms ' . 

* 

major descriptors, reducing-the collec^tioil' to 9 , 179, docinnents. 

• - . ^' ■ " ^ H ^ ' 

Next we identified- ±11 documents related to methodology using the 



Evaluation Methods. , • ^ ^. 


. (6,689) 




Research Methodology ; ^ 


(6,468) 




•Methods . 


(1,474) 




X^c^ntific Methodology , 


(492) 




Inductive Methods ' 


(47:8) 


^- ^' • • . 


Methods Research 


(344). 




- Ded^^ctive Methods / 


(272) 




This collection totaled 15,506 documents 


(iriclud^ag 


both major and minop ■ 



descriptors) . ^ ' ' ' ^ ' ' C) 
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^ la order to. identify^hose documents related to 6he methodology of 
program ^valuaftion, we "tjien 'crossed the 9,179 program* evaluation documents 
. with the 15,506 methodology documents and thus constructed a core collec- *' 

tion of 1 , 734 documents with major desc;riptors >af program evaliRation and 
,\ inajor or> minor-^tiescriptors of methodology. We were successful., therefore, 
^ in isolating a/^iore collection of c^oci^ents that numbered less^han 2,000. 
. 1,734 abscr|act;s^ are still a lot to review so we tried^to use the ERIC 
system to characterize the nature of this ^ore cqlle^tio% It would ha^>e 
been most useful if the 'ERIC system coui-d have siinply printed a frequency 
•list of the descriptors used Igthisytore c.oliection, but the system'^'ha^ . 
no such feature. -I^stelad, 1^ necessary, to choose 'likel J terms and 



inquire how many of th^ co--ce collection' ased. such terms as descriptors. ^' 
.Thrqjigh such a strategy 'tsJie following Table 1 'was constructed. 



Insert Table, -1 about here 



Table I'entries-tot^l^o 1,948 and not ;,734., sincfe the subcategories 
are nW mutually exclusllBP^able 2 indicates the overlap between these • 



subcategories. ^ 



r ... 



Insert Table 2 about here 



Notice^hat the two largest subcategories, "Data" afid /'Measurement'* overlap 
considerablys ^208 (23.4%) of the total 387 -document&irare in" both cate- • 
series) while /'Design" a^id. -'PxDlicy", among the smallest . subcategories , 
hardly overlap at .all ("2 (1.6%)" o€ the total 124 documents, are shared). 



■ : Table 1 . - * • ' 

•^ubcategorizatian of Program. Evaluation Methbdology 
• . Core Collection^ * ^ 



The- Limitations 
5 



I 



\ Subcategory 'Descriptors 

yftf^T^Policy. . / 
\ Legislation 

" ■ ■ ' ' • ( 

2. Thfoirles ^ . 
"* ; Models . . 

3. I^ign 
Research .Design 
Design Need&.^ ' , i 
Effesign Preferences - 

4. Datas ^ 
Data Gollect;ion 
Data/Trocessing 
Electronic Dat-a *Pr.6cessing 
Statistic4l Da^ta <' ^ 
Analysis 

.. '..(. 

5. Me^suremenf" . ' 
Educational Assessment ' 
NieBds Assessment - 

T^st 



^ }: 



Number -of Documents^ 

5^- • 



6. . Skills 
. ^Ethics 

Standards ^ 
Qualifications 
Credentials 
' Job ^Skills * 
Research Ski^s 



7, 
8. 



Training ■ , 

■ Management 

Program Administration 
Administration 
Administrative Principles 
Administrator Characteristics 
Administrator Guides 
Administrative Responsibility 
. Administrator's Role 



'9. Problems 
Needs 



241 



f 



71 



407 



688 

) 



>3 



128 ^ 




203 



cm (71) 


(^07)^ 


5. "HutfUiiruluuiic" 6. 


(Al) > 




(I2fl) 


1. "I'ruliltimu' 
f20)) 




12 






r — 

2 * ■' 


" '3 


6 






^ 56 


ao 




6. • 
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36 
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28 
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Only 1,245 documencs of che cocai 1,734- in the core coUecCion ol 
prc^gram evaluation methodology can* be olaced in one or more of the 
subcategories listed in Table U leaving 489- dajjuments., or 23 percent of 
the collection, uncategorized . Upon examination of a sample of these 489- 
unclassified documents, we found no other 'dominant descriptors rel^evant 
-f-or-'-our -ptirpo-ses. Instead , we found descriptors which "indicated either 

educational level or - document ■■■ro-prc"""''a:re"a*; To r""~^xampl"^ -yjx^-r^^;^-^''- - 

descripcored ' Elementary^' or "Secondary Education^' 135 vere descrip^ed 
"Vocational*. " Of che^pro^ram evaluacion methodology callection of V,734, 
1,^46 could be subcategorized according to che'se cvo c:/pes of descri'f/to^r? . 

Our actempc t:o characterize rh,e, 1,734 core aollection of program' 
evaluation mechodology documencs vas ch.erefore pnly oartiallv successful. 
Only 72. .percent of the documents could be placed in subcategories useful 
in our search, leaving 439 abstriccs to 'be screened by hand.- Not all 
documencs in che core connection had been doscr ip c^red in jufficienc 
detail for our^ purposes. 

: t ■ . "•■ " ' . ' 

Alchouah we did achieve a part,,j.a'i :)r 'il.iminary ocreening using ERIC j 

incexa^ccive corppucer ar»arching. Ic vas* :iccomplished onlv chroui^h' crial 

and error. SRIC provides no direc,c means ot charac Cei'i^iing a collacc^ion 
♦ 

ot documencs. Hven learning cht? froquoncv <.)t: corcain dose r i:: cor :i in i 
collection has Co be donJi^v :t:ervin^ one de b*<; r in c ive ac a :i:T^.e. 
FurthetTnore , onlv '^8 iocamonc Tianioulacion -sr-eps ^ro .illoved per run, 
onlv o-ne documenc :ile ^av b*"* :avpd -ic cho -^nd of a run. ind onL*'' 
600,000 documencs radv bo Lnqtiirod ibouc In ono run. ;';Rrr: prG:,;oncL'.* 



^ ' • The Limicaciotis 

coricainSj^ approximacely - 316,000 documencs, and since ^ learning • chac ■ 
34,949 c^ocumencs are djescripcored Higher Educacion conscicuc^s an inquiry 
of 34^949 documencs, one cannoc even cheo'k'eac^ document in Che syscem 
cwicfev These de3;i.gn conscrain^s greacly hamper actempts Co use che 

r 

ERIC S'^scem in an inceraccive mode on large searches — Che searches for' 

■which - it" 'i^<OT3S;t hB^ ■ ; . . ■ j---: - . ... ^ v.;v/.. V,-:;::::: 

One addiciohal feature of, ERIC should be mencioned. Beginning in 
Sepcember of 1974, all documencs entered -inco ERIC vere Co be idenCified 
by one of 19 "document type" codes such as; 



3 Books ; Monographs, Textbooks; Programmed TexCs; ^ " 

C Curriculum Guides; Teacher-Developed Mace'rlals;' 
LaboraCory Manuals; 

Journal. Arcicles ; Serials Per iodit:als ; Sullecins; 
NewsleCCei:;s ; Newspapers; 
K Program/ProjecC Descriptions; 

R Reports (Research and Tec:; • i cal ) ; (ERIC, 1974) 

Out of our 1,734 core documents ,. 680 dpcuments (EDs) and 232 journal 
articles (EJs) , for a total of 9L2, vere entered after September of 1974 
However, document type codes were available for only 676 percent) 
of thesti .91J 'Entries. Apparentlv there is some ovs cematic 'bias in 
these omissions,, since 'of the 676 coded, 367 were reports (R) ^ut only 6 
were journal irticles (J)~a quick check easily verified that far more 



* 



( 
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than .6 journal articles entered in .ERIc' since '-1974 were included in our 
core collection. The high percentage of omiss^ions and the probable bias 
dissuaded us from using the document cype codes to characterize our - 
-cor6 collection, " J 

In conclusion, our attempts co use ERIC* to characterize and initially 

collection met with limited success. But 
perhaps we were asking more' o^ the ERIC system than was reasonable. The 
system was designed primarily for storage and retrieval of fugitive 
educational do^iuments. We could still hand screen 1,734 abstracts if 
necessary, assuming the collection really contained the core of the 
literature we wished Co review. 

Identifying Literature Using ERIC 
How well did ^ur collection of 1 ,734 abstracts cover the existing i 
literature on program evaluation nrethodo ? We had some doubts about 
its comprehensiveness, not only because we had already ruled 6ut a lot 
of apparently related literature, but primarily because one cannot perforni 
hierarchial searches In ERIC. The ERIC Processing Manual warns coders to 
match the level, of the descriptors co che specific level of che document 
and to avoid descripcors broader or narrower than che scope of che document 
itself (ERIC, 1974, p. 218-219). Thus a very specific document on che use 
of case study methods in evaluation will be descriptored wich che narrower 
term "Case S Cudy" ' rather chan che broader cerm "Evaluation Methods." A 
search using "Evaluation Methods" will therefore pull only documents des- 
criptored at that broad level and will noC pull documents descripcored 

# 

at che more narrow level of '*Case SCudy," in spice of che face 
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chat such documents may discuss case scud.ies as *an evaluacidn mechod • 
in ERIC, a. search employing broad Cenis will noc identify all ./^ 
relevant documents ^descriptored using more specific terms; i ..e • ,^^^^RtC 
documents are not descr^ptored hierarchially 

An external criterion- w^s needed to test the quality of our core 
literature collection and the recent evalu^tign of .th^s 

Evaluation Center at Western Michigan Univers;i^£y" was selected (Bunda, 1976), 
Tnis bibliograp"hy contains ^603 ref erencels^-^n evaluation and is one of the 
most current and.^comprehensive bibl^ojgf aphies available. This bibliography, 
is not limited to program evaliiatimi methodology' (and we know of no such 
narrow bibliography currently available) but it is "...unique in its 
calculated inclusion - of fugl/cive refer^.iftces and a variety of resources" 
(Bunda, 1976, p. iv) . % Therefore , mg^ so than some of the previous 
evaluation bibliographies which fqcused on the "classics," this bibliography 

- . . ' / ^ / 

emphasizes ru^itive documents jvxst as ERIC does and so should serve as an 

. / - ■ " ■ '--^ 

interesting crxteri^on measure, ^ ' ' 

I 

i. Using the Current Tnjdex to Journals in Sducatj^on and the ERIC Ed 
Accessions File ^itle Index'^ ( throu^gh December 1977), each of the 603' 
references were checked tq see if thev were contained in ERIC. This infor- ' 

' / ■ j ■ ^ ' 

mation told us/ the maximum possible overlap between the Western Michigan 
bibliography of 603 item^ and our ERIC generated ^collection of 1,73A items. ' 

The results were startling. Of the 603 references, only 161 (27 
percent) werej listed in ERIC — less than a third of the Western Michigan 
bibliography pould be reproduced from the' ERIC ^ystem. Further investigation 



The ■Limd.tations 

in4icated" Why the overlap was so small. Of Ghe *603 references : 
^ 73 (12%)' wfere- journal articles publislj^d prior to 1969 — ERIC 4id 

not begin including journal articles in its collection 

_ •> ' 

lintil' 1969,'. 

83 (14%) were books published prior co igee-.-ERlC did not exist until 
1966. 

\i (v 2%) were' fugitive, reports wtitp^n prior to 1966 — ERIC did not 

) ' ' ^' ' ' ^ 

exist until 1966. 

Therefore, 28 percent of the Western Michigan bibliogra^j^hy was published or 
written before pRIC was incorporating such documents into, its system. But 
what about the remaining 45' percent of the bibliography which is not in 
SRIC? ' ' ' , 

132 (22%) of the references are books or chapters in books published 
between 1966 ■ and 1976^ ' Because of copyright and reproduction restrict 
tions^ ^commercial publications are no.t i^^tJ^lly made available through E^IC ' 

In fact, ERIC tries to keep the number of documents it*"cannot reproduce 

I ^ I ■ . ^ 

down t(^. 18 percent of its /total collection (ERIC, 1974, p. 36). Thus books 
and' oth^r publistied raateriials, chough current and relevant, will not be 
found in ERIC. -This results in major omissions; for example, neither the 
AERA nor Ghe CSE evaluation monograph series are rq^erenced in ERIC. ' 

56 (9%) of the omitted Western Michigan references are journal articles 

published between 1969^1976. Since ERIC included all major journa^Ls, how 

/ ' . 
were th^e articles missed? The answer is that ERIC, in fact does not 

referejce. the complete contents of all major journals. The Current! Index Cp, 

Journals in Edu(tation July-December 1977, issue lists 739 journals that were J 
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screened during Uhat period. However, only Ib'j journals (25%) had all 
df^their art 1(^16^3 '^entered into ERIC; for the remaining 554 journals, 
Only selected^^^ticles were entered. Most journals are a''ssigne4 to 
iifinghouses which decide how. muLh of ^ journal ' s.-..i 
content to eqtkr. - Thus if an evaluation article appears in a journal 
assigned to 4 Clearinghouse other than ERIC TM, it may not fall within 
the interest/s of that Clearinghouse and wiir'either ' simply ""'hot """Be"' 

1 selected, o/r at most referred to ERIC TM. An ERIC search, "therefore, does 

/• * 
not assure /one of covering . all relevant journal articles. 

86 (ij^Z) of the , omitted -references were fugitive documents and 

reports vtitteh dur:^g 1966-1976. These documents should , have been 

entered into ERIC according to the system's guidelines^ Either" 



the ERIC^TM Clearinghouse never became aware of the existence of 
these dcj'cuments ,or - t^ey did not meet ERIC * s. selecjp'ion criteria of reproduc- 
ibility !anA importance. _ ' 



T^e 161 docljmfents of the 



estern Michigan bibliography which were in 



C'Were of tfte following types': 
naS 



i'M'^ " ^3 were journaP art?icles published betw^n 1969 and 1976 , 

^ ^|p<,wer^d^^ or chapiters in books publish^^^i prior to 1966;, 

^brvcrha^ters in books published between 1966 and 1976, 
'tM J^^^^^S^If^*^^^ 1 m P n r q writLeu pi lor to 1966, and 





ttoduments written ^between 1966 and 1976. 



To deDemine the overlap between our 1,7 3^ core documents an<l the- 



• -^3*^estern Michigany^ibliography , the ED or EJ number of each of the 161 

'irij^'f erences from the bibliography was checked against the .1,73^ ED and EJ 
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numbers of che cQr^ 'collection. The overlap curned ' ouc "^ro oe an 



astouhdirig 7_ documencs of which: ^ 

wjere journal articles published 1969-1976,- 

were books or chapters 'in books published 196*6^976, and 

2^were fugitive documents written 1966-1976. 
% 

Why were 154 documents -^of the bibliography nor picked up/in our 



.search, even thoegh they were. ^ in ERIC?~""~K''examrnat^^ 

documents and their descrip tprs uncovered four general reasons. 

First, 'some items^in the bibliography were not directly related to ^ 
"the methodology of program evaluation but focused. on such topics as 
research ethics and standardized testing. 

Second, some^^^tems related to specific evaluation methodologies such 
as the Delphi and critical incident ^ techniques'. , These were not selected 

in the 'search because broader descriptors such as "Evaluation Methods" 

>^ 

had been used. ' ' 

Third, some descriptoring , although done-^S^ accordance with ERIC 

I * 

guide|.ines, makes identifying certain dQCuments problematic. Fb^r example, 
{a) Scriven's chapter "Evaluation Perspective^ .ad Procedures') 
has "Summative Evaluation" as a major descriptor, while the 
book it -appears in, Popham's Evaluation in. Education; Current 
. { Applications (1974), does not, * 
(b) Wright and Worthen's report (1975a) on evaluation contracting has 
different descriptors than the^summary of that report (1975b) 
which is also in ERIC (t.he report and summary were descriptored 
by different coders) , and 
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^c) A design developei by Stuff lebeam et .al. (1971) for 

evalij^Cing R&D institutions and prog^m's is 50 1' -d^criptored 
under 'J^valuation Methods'*, "Methods Research", or "Methods", . ' 
while a simila::, design developed by Scriven et al . (1971^^^ 
■ dB^cripcor^. "Evaluation Mecho4s". ' These two repoxtsywere com-'^^ x^ 

missioned by Che Office of Education at the same z±Ae for 'the same 
pu|^se; they w^re . ent>^red into ERIC by the same coder, but 
"^^ontain diff erent-descriptors. 

■ ■; " -.-A... ^. : . ^ , J. i„__.. 

The practice of descrip coring 'only Che documenc in hand makes ic possible 
that two highly related documents iiayf have\^f f erent descriptors and not 
be retrievable under Che same search ' strategy . / ■ ^ 

Fourth, Che assignmenc of descrip cor s obviously .depends on the backr 
ground, experience, and poiiic of view of che coder; To see the relevance 
of certain c^ocuments to program evaluacion mechodology, or any other 
specialty, requires considerably more familiarity with the . fiiaj.r^ ch-nn 
chat possessed by most . graduate students who do che mai^ 
house coding ft-^ TC Manual does recommend :hac at ,iec .-uuuCenc 
specialise partx ,jace in che cloding/ of * encries) 

'Wiile the J.ack of overlap between the Western Michigan bibliog^phy 
and the 1,734 cdjre program' evaluacion aiechodo^gy liceracure coll^cion 

r ' 

can be explained in c^rms of che deficiency of che bibliography as a 
proper cricerion measure, che desigil constraints of che ERIC 'descripcoring 
system, and human limitations, che trivial overlap is cfercainly noc 
reassuring. 
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^ . • Conclusions 



^ . . - ♦ 

*Larg'e, integrative literature reviews are becoming increasingly • 

needed and. increasingly difficult due to the rapidly expanding llrt/erature 

in education. We have attempted to use the EM^C system , not only to identify 

relevant literature buT^also to screen it.' -Because of the lacic of . 

s^ufficient depth of descrip toring, it was possible to subcategorize only 

72 perdent^^f^ our initial document collection. ERIC currently provides 



EKLC 



. no mearre to charac terize ah existing "do ^ - - 

system's "document type" code facility was available for only threrf-f our ths 

^ , \ ■ ; ( 

of the documents it was desired to eover. While initial screening of small 
. document collections is possible, sy^sCem design constraints prevent complex 
screening or screening large document files. 

eric's lack of'Succ^sfe in iJdentify: ajor evaluation documents is 
a,-morei troublesome, bud understandable when on^^^ealizes that 
\(a) no jpurnals before 1968 are in ERIC, 

(b) only approximately 25 percent of major education journS 
^ fully covered by ERIC, 

(c) few fugitive documents and books prior to 1966 are in ERIC, 
(4) few books or other maj(^r published mater^ls are in ERIC 

(maximum of 18% of the system), and 
(e) only fugitive materials identified by the specific 
Clearinghouses of intere^'s-t are in ERIC. 
The added problems, of non-hierarchial searching, system descriptoring^- 
cqnstraints, and codef limitations 'forces one to a conservat;ive view of 
the products of an ^IC search* | 
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The results of the foregoing s^a^h can,' of course, not be^ freely 
generalized' -to other subsets of literature with ERIC. Howeve^ several 
of the* problems we have identified are«in'fa^t due to system limitations* 
and will obtain regardless of the- descriptors being usecf. <^ % 

With the growing literature in education we are forced to use such 
systems as ERIC, but we must fullV understand , their ^imitations'. With- 
small, fpcused search-es it may be possible* to identify comprehensive sets 
of documents and review the absi^cts by hand. "J^^ith larger 'searches , ^ 
)re^s6phisticated , automa ted' screeriirie capabili is needed than not^r 



more^ sophisticated, automated screening capability is^^.u^^ 
exists in ERIC. Until such a capability exists, researchers will need to 
augment ERIC, searches with the more traditional lite^ture identif icatig^n 
and search procedures. ERIC is currently not an adequate iblution to our 
problems of identifying and characterizing the burgeoning literature in 
edufation. \ • * 
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NpTE 



''/^isji express my .sincere . thanks t-o Donna Shaver who j^ollaborat^d 
on the computQ<&2ed ERIC searches and to Denfise Sniith who assist 
with several clerical and editorial ta^ks. ^ . ^ ' 

.^The searches' described here were conducted in mid-May of 19?3 and' 
UovQT^d all hllC entries from its-'lnception in 1966 to that time. 
The searches ^wfere conducted usir>g the Lockheed Infoinnation System* s 



Dialog System . Al though DialbY is tSe^ larges t 

system, an international system accessing approximately 75 d^Sa basds, 
its design, is not necessarily representative of other access ''sys t ems . 
Thus the computer ^access design limitations discussed in this paper 
pertain only to the Dialog System as it i| now instituted. 
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